Summary:

X-link: facebookresearch/FBGEMM#2647

Context
---------

Running MPZCH on AMD hardware causes a CPU-to-GPU hang. This was identified to be the ZCH GPU kernel during the spin-wait loop that occurs.

TorchRec creates metadata and sets its default values to -1. This is then provided to the FBGEMM kernel. Note if an identity slot is -1, then its metadata is also -1, and the metadata for that slot is never -1 again when it gets updated.

Suppose thread A writes in slot i, and updates the identity slot with its item.  Suppose thread B was late and is also trying to write in slot i. While the metadata is being updated from thread A, thread B runs a spin-wait loop waiting for the metadata to be updated. This creates many atomic calls just to see if the metadata was updated from thread A.

AMD MI3X handles atomics differently than NVIDIA GPU. Due to this, the latency is incredibly slow, and atomic contention can occur when many threads are reading the atomic value . This ultimately looks like a hang that occurs.

NOTE: this only impacts when identities are first inserted when their metadata is -1, and explains why the hang occurs during the first few batches. WIP to reducing the amount of atomics in ZCH.

Implementation
------------------

Added a spin counter (SC ) to the spin-wait loop in check_evict and check_min. The value is determined through benchmarking

Reviewed By: kaanbaloglu

Differential Revision: D102424864